Abraham Lincoln came from the humblest of beginnings. His wife Mary Todd hailed from a wealthy clan. Their lives together were marked by personal tragedy.

Lincoln’s mother, sister and three of his sons died young. Losing the boys crushed Lincoln and Mary, who also lost two half-brothers and a brother-in-law during the Civil War. Here’s a look at Lincoln’s family, Mary’s Confederate relatives and tragedies that wracked the family:

Thomas Lincoln, a homesteading farmer and sometime cabinet maker, chose to raise his family on the harsh Midwest frontier. A stern father who was likely illiterate, Thomas never fully understood Abraham’s desire to further his education and reprimanded his son for reading instead of tending to chores.

Thomas never met his daughter-in-law, Mary Todd, or his grandchildren. Abraham, busy with work and a sick wife, did not travel to attend his funeral.

“In all of his published writings, and, indeed, even in reports of hundreds of stories and conversations, [Lincoln] had not one favorable word to say about his father,” wrote Lincoln biographer David Herbert Donald.

A Virginia native, Nancy moved to Kentucky, where she married Thomas Lincoln and gave birth to their three children: Sarah, the eldest; Abraham, the middle child; and Thomas, who died in infancy.

Lincoln called his mother, a tall, slender woman with black hair, “highly intellectual by nature,” with a “strong memory” and “acute judgment.”

In 1816, the Lincolns moved to southern Indiana and constructed a small cabin on Little Pigeon Creek. Two years later, Nancy died from “milk sickness” or tuberculosis.

After Nancy’s death in 1818, the burdens of keeping house fell to Lincoln’s 11-year-old sister Sarah. Like her brother, Sarah—who went by “Sally”—was intelligent, had a keen sense of humor and a gift for putting people at ease.

When 19-year-old Abe received news that his older sister had died in childbirth at the age of 20, he buried his face in his hands and sobbed, his former law partner and biographer William Herndon wrote.

In 1819, Thomas Lincoln returned to Kentucky to propose marriage to Sarah Bush, whom he’d known earlier. Nearly 10 years his junior, Sarah accepted and moved with him to Thomas’s Indiana farm.

Suddenly, Abraham had three new half-siblings: Elizabeth, Matilda and John. Sarah recognized young Abe's intelligence, encouraged him to better himself and the two developed a close lifelong bond.

“The quality of warm human kindness so marked in Abraham's character was a reflection in part at least of his happy home life as a boy after Sarah became his stepmother,” wrote her biographer Charles Coleman.

Born into a large, prosperous Lexington, Kentucky family, Mary Todd Lincoln lost her mother at age 6. Her strict stepmother later sent her away to school, where she received an elite education, studying French and the humanities.

In 1839 in Springfield, Illinois, she met Lincoln—“a poor nobody then.” Three years later, after a stormy courtship and a broken engagement, the couple wed on a rainy Friday before about 30 relatives and friends.

Mary, an avid follower of politics, shared in the successes and misfortunes of her husband, who took national office for the first time in 1847, as a U.S. representative from Illinois. After Lincoln was elected president in 1860, she served as de facto White House social director as well as Lincoln’s confidante.

“He was often filled with gloom and despondency which it took all of Mary's adroitness to dispel,” wrote her half-niece Katherine Helm in a biography of the former First Lady.

Mary fought her own battles with mental health—and crushing grief. In addition to losing her mother at a tender age, she lost three young sons and witnessed her husband’s murder at close range. After leaving the White House, she struggled financially and endured wide condemnation for her erratic behavior, which during her lifetime was never understood as possible bipolar disorder or acute post-traumatic stress. In 1875, a public trial initiated by her son Robert declared her insane and committed her to a sanitarium. Finagling a release a few months later, she moved to Europe, where she lived until a year before her death.

Lincoln’s eldest son—the only one to live to adulthood, marry and have a family of his own—left an impressive legacy. A graduate of Harvard’s class of 1864 and an officer on Ulysses Grant’s staff in the waning days of the Civil War, Robert later served as Secretary of War under two presidents and a minister to England.

Studious and inquisitive, Robert enjoyed astronomy and working on algebraic problems. When his health declined in his later years, he took up golf, telling friends the sport saved his life.

“He was of a taciturn and retiring nature, and only to his close friends did he reveal himself as a charming conversationalist and an entertaining storyteller, a trait which he inherited from his father,” read an obituary.

The cries of a grief-stricken Mary Lincoln echoed throughout the family’s house in Springfield when 3-year-old Edward died, likely of tuberculosis. The Lincolns’ second child was named for Edward Baker, a friend and politician who would become a U.S. Army officer during the Civil War.

Edward originally was buried in Springfield. In December 1865, his remains were disinterred and re-buried in the Lincoln Tomb with his father at Springfield’s Oak Ridge Cemetery.

Willie Lincoln could be rambunctious like younger brother Tad—who was known for pulling pranks in the White House—but he was also studious and thoughtful. In October 1861, following the wartime death of Baker, the 10-year-old submitted a poem about the soldier to a local newspaper.

In February 1862, Tad and Willie fell ill with typhoid fever. Tad recovered, but Willie died, devastating his parents. “My poor boy, he was too good for this earth. God has called him home,” Lincoln moaned.

Following Willie’s death, Mary visited a spiritualist, who she said made “wonderful revelations” to her about the dead boy. Lincoln worried for her mental health.

But Lincoln's secretary John Nicolay downplayed reports of the first lady’s odd behavior after Willie’s death. “That she was very eccentric there is no doubt,” he wrote in 1887, “but [that] she went to the extremes reported I do not believe.”

Lincoln nicknamed his youngest child "Tad" because he was “wriggly as a tadpole" as a baby. As a youngster, Tad spoke with a lisp, probably because of a cleft palate.

Impulsive and mischievous, Tad was “idolized by both his father and mother, petted and indulged by his teachers, and fawned upon and caressed by the noisome horde of office-seekers which infested the ante-rooms of the White House,” wrote John Hay, Lincoln’s assistant and secretary.

“The tricksy little sprite…gave to that sad and solemn White House of the great war the only comic relief it knew,” Hay wrote.

In the summer of 1871, 18-year-old Tad fell ill after returning from Europe and died in Chicago of unknown causes.

Mary’s brother George R.C. Todd and three half-brothers (Alexander, David and Samuel Todd) all served in the Confederate Army. Samuel fell at Shiloh, Alexander at Baton Rouge, Louisiana. David was wounded at Vicksburg, Mississippi.

Southern newspapers castigated Mary, a native of border state Kentucky, following Samuel’s death on April 5, 1862. “It must be a pleasant reflection to Mrs. Abraham Lincoln, amid her vulgar attempts to ape royal fashions, with her balls and soirees at the Federal capital, that a gallant brother should have thus fallen by hands of her husband’s mercenaries,” a Louisiana newspaper noted.

In September 1863, Mary’s brother-in-law, 32-year-old Confederate Brig. Gen. Benjamin Hardin Helm, fell at the Battle of Chickamauga (Georgia). Married to Emilie Todd, Mary’s younger half-sister, Helm had declined the president’s offer of a position in the Union army.

"I never saw Mr. Lincoln more moved," said Illinois Senator David Davis of the president’s reaction to Helm’s death. "…I found him in the greatest grief.”

In December 1863, U.S. authorities granted Emilie passage through the lines to visit the White House, where the Lincolns treated her with kindness during a six-day visit. "Mr. Lincoln and my sister met me with the warmest affection, [but] we were all too grief-stricken at first for speech," the 26-year-old widow wrote in her diary.